feat: add an option to skip validate lance table#239
feat: add an option to skip validate lance table#239hfutatzhanghb wants to merge 3 commits intolance-format:mainfrom
Conversation
|
ACTION NEEDED The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. For details on the error please inspect the "PR Title Check" action. |
hfutatzhanghb
changed the title
|
Hi, @jackye1995 @wojiaodoubao could you please help review this PR when you have free time? Thanks a lot. |
|
“t2 is not lance format table” This seems like a valid error since the table property |
Table t2 is a non-lance format table. We just attempt to load data from non-lance table and insert those data into lance table t1. |
|
I think there are 2 problems here, (1) INSERT OVERWRITE is for Hive only, for Spark DSv2 you should use CREATE OR REPLACE TABLE AS SELECT. (2) lance sits in its own Spark catalog, so you need something like |
2 participants
Motivation
Recently, we are testing lance-spark with hive2 or hive3 namespace. We met an problem as below:
suppose we have two tables: t1 and t2. t1 is a lance table and t2 is a normal hive table. It ocurred exception when we execute below SQL:
The output exception is
t2 is not lance format table. After refering to the code, we found it it blocked by methodvalidateLanceTable.Solution
We should better add an option to control whether we should skip validate lance table or not.